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ABSTRACT 



The United States is transitioning to an information age 
society, requiring that the citizens and workers of the future be highly 
literate in their use of information. Information literacy is the ability to 
evaluate, analyze, and apply critical thinking to the use of information. 
Educators need to answer basic questions about how students become 
information literate using online resources and other technologies, which 
online resources best promote information literacy, and what strategies best 
prepare teachers to facilitate this learning. Performance assessment can 
provide a reflective process to inform these questions. The first step is to 
agree on a conceptual base. Assessment begins with the background each child 
brings to school and ends with competencies all high school graduates should 
possess. Performance-based tasks, longer projects or experiments, as well as 
student portfolios must be considered as assessment strategies; a progression 
of strategies that are developmentally appropriate for grades K-12, including 
group based projects as well as individual tasks can be developed. It is 
imperative that the assessment strategy specifically supports national 
initiatives. The development of information literacy assessment will involve 
teachers, administrators, subject area experts, students, and key project 
staff, all under the guidance of the Assessment Coordinator. The major steps 
in developing the performance assessment for information literacy are: 
defining the constructs to be measured; defining the target population; 
reviewing related tests; developing a prototype; evaluating the prototype; 
revising the tool(s); and collecting data on test validity and reliability. 
Professional development is critical to success; teachers are the key to the 
attainment of information literacy. (Contains 15 references.) (AEF) 
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Abstract 

The United States is transitioning to an information age society, requiring that the citizens 
and workers of the future be highly literate in their use of information. Information literacy is the 
ability to evaluate, analyze, and apply critical thinking to the use of information. Evaluation of 
information literacy is a construct that is being developed by default in practice. While grounding 
in practice is immediately practical, we also need to work for long term goals. We need to answer 
basic questions on how students become information literate using online resources and other 
technologies, what online resources best promote information literacy, and what strategies best 
prepare teachers to facilitate this learning. Performance assessment can provide a reflective process 
to inform these questions. Three major information literacy products need to be developed: (1.) the 
assessment tools, (2.) professional development models and training materials, and (3.) on-line 
information curriculum linked to school reform Frameworks and Standards. Assessment, as a 
formative activity, is a way to start. To do this, parts of traditional tests and measurements, other 
related but less central measures, such as those of critical thinking, and classroom-based assessment 
all need to be represented in the evaluation discussion. The purpose of this session is to provide 
rationale, discuss possible recommendations, and start a focused national dialogue on performance 
assessment for information literacy. Assessment tools for the construct of information literacy are 
absolutely necessary to establish credibility and to shape the development of practice in education. 

U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and improvement 
EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

□ This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



fax 805/636-4135 
cdoyle@kem.org 

Robert E. Holloway 



"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



Contact information 

Christina S. Doyle, Director of Learning Technologies 
Kem County Office of Superintendent of Schools 
1300 17th Street 
Bakersfield, CA 93301-4533 
805/636-4331 



O 

RIC 



BEST COPY AVAIUBLE 



2 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)." 



1 

Performance assessment for Information Literacy 

Performance Assessment for Information Literacy 

School budgets depend on public support, and public support 
depends in part on public perception of school effectiveness. 

John Hoven 



Information literacy is the ability to access, evaluate, and use information from a variety of 
sources. Information literacy is best thought of as a verb, a way of doing information. The construct 
is not fulled developed but it is easy to see a conceptual sweep that includes basic skills in reading 
literacy, the mechanical skills of using technology, critical thinking, and problem solving. We, those 
advocating information literacy, are working on two basic questions. 

► How does one become information literate? 

► What practices work best in teaching and learning to be information literate? 

Assessment is an important part of the process of answering these questions. The reasons 

are compelling: 

Public and political stakeholders demand credible evidence; 

Practice is shaped by assessment. 

These two reasons are natural occurrences and affect aU practices. The first reason, credible 
evidence, is a matter of perception, an admittedly constructed reahty that has real consequences. 
That school budgets depend on the public’s perception of effectiveness is obvious. That political 
decision makers depend on public weald is also obvious. This is more a natural law than a moral or 
philosophical position. The political pressure to make decisions based on standards and traditional 
appearing measurement is great and increasing. We must have credible evidence of outcomes. 

The second reason has two forces shaping practice. One is the kind of decision teachers 
and employers make in selecting and continuing iimovations. “Does this process offer a relative 
advantage?” Or, “Does it fit with existing practice?” In short, what is the advantage of teaching 
information literacy? The other force is external evaluation, the shaping of practice in response to 
standardized assessment — a powerful determinant of practice. This paper is to encourage 
reflective and intentional development of assessment of information literacy in a milieu where 
assessment is unavoidable. 
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Assessment can have negative effects on the development of practice, stultifying the 
conceptual spread of an idea and making practice rigid. James Carey (1997) suggests procedures 
for measuring students' attainment of higher order-outcomes in information skills. Christina Bmce 
(1995) has reservations about measuring the phenomenological event of information literacy. Carey 
is rightly concerned about these skills and Bmce’ reasons for avoiding standard measures is 
appealing. However, in practice decision makers’ judgments of worth will be made regardless of 
philosophical position or intent. 

The California experience with the CLAS assessment process demonstrates what happens 
when the public and political agendas are not addressed. The tme challenge is providing 
appropriate assessment. Our worst examples of this are assessments of information literacy that end 
with finding information as the last step, not using information in real application. Information 
literacy is not an academic exercise. 

The Secretary’s Commission on Achieving Necessary Skills (SCANS) Report (1992) on 
employer needs in the workplace make the real world implications clear. Demonstrating economic 
consequences is only one subset of assessment but it is heavily weighted in public discourse. So, as 
tired as we may be of “real world” as a way of determining worth, it is a challenge we must not 
ignore. If we do, information literacy will be marginalized as an innovation. 

Background 

Information literacy in telecommunications is achieved when learners know when to use 
on-line resources, how to access information competently, how to evaluate information for accuracy 
and significance, and how to use this information to communicate effectively. Learners who are 
able to do this have a life long skill to meet the challenges of the information age. 

The concept has been disseminated by the National Forum on Information Literacy , a 
coalition of over sixty national organizations from business, govermnent, and education, all of 
which share an interest and concern with information literacy. It was organized as an outcome of 
the American Library Association's Presidential Commission in 1989, and has met quarterly since 
then to promote the concept of information literacy as imperative for the current Information Age. 
The purpose of this umbrella group is to disseminate the concept to other professions. 
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Information Literacy . Information literacy is a basic survival skill for the 21st century. It 
is not a “school” skill but one needed in aU settings. It can be summarized as the ability to access, 
evaluate, and use information from a variety of sources. Significant advances in and need for 
information literacy are, to a large degree, possible because of the new technologies. 

As classrooms move from a reliance on a single textbook to multiple sources in print and 
non-print formats, students are nudged away from finding the "right" answer (printed in a text), to 
finding the "best" answer from aU available sources. The goal is not to find information, a sort of 
academic scavenger hunt, but to do something practical with it. "What is it I really want to 
know?" must be answered clearly before one can decide where to search among resources. 
Clarifying what is already known is helpful; so is recognizing that in this process, the question 
should change as information emerges. Information literacy can be achieved when schools are 
restructured around resource-based learning. Resource-based learning requires restructuring of the 
learning environment, the learning process, the role of the student, the role of the teacher, and the 
relationship between student and teacher. 

The critical need has now become looking at what differences this telecommunications 
resource has made with the learners. Have students become information literate — able to use 
information productively and creatively when facing new situations? Has the transfer of learning 
been successful — are these students better prepared to be lifelong, independent learners? How do 
we provide learners with experiences that lead to greater information literacy? The availability of 
on-line resources should make teaching information literacy a priority in every classroom. 
However, these changes have happened so quickly that most individual teachers have not had time 
to review and evaluate the resources and develop meaningful lesson applications for the Internet. 
Many do not yet have the technical confidence even to attempt accessing resources on the World 
Wide Web. 

To date, much of the work of making technology part of education has been focused on 
using the equipment, and teaching people how to manipulate computer languages, software, and 
hardware. Now the new "point and click" access to the World Wide Web through tools such as 
Netscape tools have made technology truly accessible as a learning tool for people of even limited 
technical ability. The focus now needs to shift to curriculum and critical evaluation of the on-line 




5 



4 

Performance assessment for Information Literacy 

information. 

Performance Assessment . Tests and measurements represent a way of gathering information 
to make judgements. Judgements will be made with or without tests and measurements. Standards 
and testing are part of the current political and public dialogue. Sabers & Sabers say that “perhaps 
it matters less what standards are initially adopted ... than what adaptations are made during 
implementation (1996, p. 21). We are early in the process of discussing standards so adaptation 
must be the theme. Further, there are reservations about practice. Barak Rosenshine, in recent 
AERA Division C listserv discussions, noted that the instructional practices that are called 
constructivism today are hardly new. The same practices appeared in 1918 in Kilpatrick's Project 
Method, and reappeared in the 1970's as Open Education and Hands-On Science. He said he 
understood the passion that many have for these ideas, but did not see any new instructional 
developments here nor any convincing data. Reliability is a major problem. We aU know the story 
of the essay that was sent to six readers and received grades 
from F to A. That problem, he says, exists today with authentic assessment. 

The Office of Technology Assessment defines performance assessment as "testing methods 
that require students to create and answer or product that demonstrates knowledge or skills" (cited 
in Improving American Schools Act Newsletter, 1996). Performance-based tasks, longer projects 
or experiments, as well as student portfolios will aU be considered as assessment strategies. The 
emphasis is on tasks for real life solutions and communications. For education, a progression of 
strategies will be designed that are developmentaUy appropriate for grades K-12, including group 
based projects and individual tasks. 

Being proactive in assessment puts a stronger case before the stakeholders than “oughts,” 
“shoulds,” and vague philosophical statements. Different, and emotionally potent, definitions of 
assessment make assessment discussions difficult. Testing has joined the forbidden topics of 
religion, sex, and politics for those who want harmony and nonconfrontational conversations. To 
reduce some of the risk in this paper, the kind of testing advocated is grounded in low inference 
measures. This category of assessment is limited to performance of tasks directly related to the 
skills being measured. Higher inference measures, such as intelligence tests, are of doubtful value 
at this stage of development of the construct. 
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Rationale 

The computer has the capabihty of connecting our students to an incredibly dynamic library 
of resources with the potential to provide every classroom, whether in an isolated desert or a 
culturally diverse urban school, equal access to the richest wealth of knowledge available in history. 
However, this potential is yet unfulfilled. Schools may say they are building "technical skills" but in 
reahty students have not developed the information literacy needed to manage the data glut or to 
make informed decisions using this new wealth of knowledge. Students need s kills in accessing 
information, evaluating the accuracy and appropriateness of this information, and practice in 
applying the information to solve problems and to use appropriate technology to create real life 
presentations or useful products. Many of the computer resources currently in schools have not 
been used to support the real learning or the fundamental goals of education. Thornburg describes 
this as a profound paradigm shift from a teacher-centered classroom to the use of technology to 
create a learner-centered environment. If the emphasis on learning s hift s to the learner and we 
agree there are political and pedagogical reasons to measure the degree of success, then the 
development of information literacy assessments is a given. This paper is a tentative proposal of a 
framework and process. 

The First Step 

The first step is to agree on a conceptual base. We suggest using the SCANS report (1992) 
because it has gained and maintained credibihty with a wide variety of users. It offers a structure 
to sort out levels of skills in general and it can be applied to information literacy. As a reminder, 
the parts are basic skills, thinking skills, and personal quahties. An abbreviated sample in each 
category is listed below to ground the discussion. 

1. Basic Skills 

► Reading: locates, understands, and interprets written information in prose and in documents 
such as manuals, graphs, and schedules. 

► Writing: communicates thoughts, ideas, information, and creates messages in writing; 
creates documents such as letters, directions, manuals, reports, graphs, and flow charts 

► Arithmetic, Mathematics: performs basic computations and approaches practical problems 
by choosing appropriately from a variety of mathematical techniques 

► Listening: receives, attends to, interprets, and responds to verbal messages and other cues 

► Speaking: organizes ideas and communicates orally 
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2. Thinking Skills 

► Creative Thinking: generates new ideas 

► Decision Making: specifies goals and constraints, generates alternatives, considers risks, 
and evaluates and chooses the best alternative 

► Problem solving: recognizes problems and devises and implements plan of action 

► Visualizing: organizes, and processes symbols, pictures, graphs, objects, and other 
information 

► Knowing How to Learn: uses efficient learning techniques to acquire and apply new 
knowledge and skills 

► Reasoning: discovers a rule or principle underlying the relationship between two or more 
objects and applies it when solving a problem 

3. Personal Qualities 

► Responsibility: exerts a high level of effort and preservers towards goal attainment. 

► Self-Esteem: believes in own self-worth and maintains a positive view of self 

► Sociability: demonstrates understanding, friendliness, adaptability, empathy, and politeness 
in group setting. 

► Self-Management: assesses self accurately, sets personal goals, monitors progress, and 
exhibits self-control 

► Integrity/ Honesty: chooses ethical courses of action 

Several of the generic SCANS Workplace Skills are closely related to information literacy. The 
obvious ones are: 

► Identifies, organizes, plans, and allocates resources including time, money, material, 
facilities, and human resources 

► Acquires and uses information including skills of evaluation, organization, maintenance, 
interpretation, communication, and computer use 

► Understands complex social, organizational, and technological inter-relationships (systems) 
and works and operates effectively with them, monitors and corrects performance, and 
improves or designs them 

► Works with a variety of technologies including their selection, application to tasks, and 
maintenance and troubleshooting. 

Others are not part of the basic definition, such as “chooses ethical courses of action.” But this 
concern is a major strand in our practice, especially related to privacy and copyright. Our argument 
is that the skills list is more a process of adaptation than creation. 

Instruction is no longer one textbook expounded by the teacher, but a variety of resources 
from which students must extract needed information ~ in short, resource-based learning. In 
resource-based learning, students select the resources they feel will best meet their needs for 
information. Their choices of materials may not differ from their teachers, but the focus has shifted 
to the students. Learning becomes an active, student-directed process. We agree with Haycock 
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Current testing practices in American education do not provide very 
powerful tools for assessing the effects of efforts to teach thinking and 
reasoning. Testing practices may in fact interfere with cultivation of the 

kind of higher order skills that are desired. Resnick, Lauren B. (1987) 
Education and Learning to Think. National Academy Press. Washington, D.C. 



The work of developing performance assessment measures will be hard in some ways. The same 
torture of ideas that occurs in the disciplines will creep into this effort. Resnick makes the point 
about testing in general. My experience with peers working in information literacy is that they are 
very student centered. At the same time, I see some use of information literacy as a new bottle to 
hold the sour wine of card catalog-type skills. If the larger professional concern is focused on what 
is best for the students, we have a chance of affecting practice in a positive way. If we let it drift, 
such measures that are developed may well interfere with the very skills we want to teach. 
Developing independent lifelong learners requires we develop effective evaluation measures for 
student information literacy performance. From this assessment we can leam how to enhance our 
knowledge of effective information literacy training and instmction and demonstrate to others the 
value of information literacy. The concession is made that the process sounds closer to vocational 
training than the Jeffersonian ideal of the educated citizen. However, these do not need to be 
mutually exclusive. The realities of successfully advocating assessments require close attention to 
widely accepted vocabulary and paradigms. 

Assessment begins with the background each child brings to school and ends with 
competencies aU high school graduates should possess. A baseline evaluation is made, followed by 
regular assessments of developmental progress. Performance-based tasks, longer projects or 
experiments, as well as student portfolios must be considered as assessment strategies. The 
emphasis, drawn from the SCANS foundation, will be on tasks for real life solutions and 
communications. A progression of strategies that are developmentally appropriate for grades K-12, 
including group based projects as well as individual tasks can be developed. The assessment 
strategy must specifically support national initiatives (SCANS Report, Improving America's 
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School Act, Goals 2000) by providing performance-based opportunities for all students. The final 
outcome is to ensure that graduating students are prepared to enter and function effectively in the 
workforce, make informed decisions in exercising citizenship, and pursue personal interests as 
lifelong learners. 

A Proposal 

The development of information literacy assessment will involve teachers, administrators, 
subject area experts, students, and key project staff, all under the guidance of the Assessment 
Coordinator. During Year One, teams from pilot schools assist in developing and testing initial 
assessment sample activities. In addition, regional trainers will be trained in anticipation of 
expansion. This "jump start" prepares trainers to assist regional site teams in using information 
literacy assessment measures. Starting the second year, new schools will be brought into the 
development process through professional training. The training materials will be refined and a 
larger pool of test items, procedures, and ideas will be available. This will support the development 
of norms and guidelines. 

Students will offer important feedback on what is working from "inside the model." They 
will develop "portfolios" of their work throughout their involvement, including writing reflectively 
of their experiences. Student journal entries will be analyzed for ways to strengthen the learning 
experiences for all learners. Ongoing monitoring of these self-assessment portfolios will be used to 
refine the tool. 

Teachers selected for the assessment team will engage in action research projects, looking 
more closely and reflectively at an aspect of class behaviors' and7or activities. They will be guided 
through this process by leadership teams who will have structured and non-structured discussions 
online. Members of leadership teams will spend time on reflective writing. Their insights will be 
captured and analyzed as they develop fluency with analysis and find practical applications of 
information literacy for classroom learning. 

Through perhaps five years, development and testing will continue, but it is anticipated that 
the majority of the examples as well as the rubric(s) will be designed by the end of the third year, 
with additional examples and refinements the goals of in years four and five. This last phase will 
include the development of a professional development product with introductory multimedia. 
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guides, models, and assessment tools. The product will be available for national dissemination, 
with significant portions available online. The Colorado model developed by the Educational 
Media Association (1994) and available on-line, demonstrates the power of an online approach to 
development and dissemination. 

Details of Developmental Process 

The major steps in developing the performance assessment for information literacy are: 

• Defining the Constructs to be Measured. Both the process and the products of student learning 
are to be included in this assessment. E*reliminaiy steps will include developing sample tasks for 
each of the sites that promote information literacy in problem-solving or decision-making contexts; 
identifying strategies to analyze both the learning process and the final products; testing these 
through norming activities across the target development sites; determining what learning 
constructs to measure. All teachers involved with development and classroom testing will be 
trained to conduct action research projects in conjunction with the research and development of 
the information literacy assessment tool. This community of learners will add to our knowledge 
considerably as to barriers to and successful strategies for implementation. 

• Defining the Target Population. In the first year, pilot sites will be selected. In subsequent years 
additional sites will be selected to work with developing and testing the tool(s), covering grades 
K-12 in wide and varied learning settings. Teachers from these sites will collaborate on developing 
and testing these assessment measures under the facilitation of the leadership teams and other 
experts as needed. Action research strategies will be employed to gain important insights from 
illuminating experiences, and group discussions online will extend the learning community 
throughout the project. 

• Reviewing Related Tests. A growing body of performance assessments inform the development 
process, although the development of this information literacy measure will require innovation. 
There are several national associations actively working to develop projects integrating 
information literacy concepts with high content standards. Among these organizations are: the 
Association for School Curriculum and Development, the Coalition of Essential Schools, the 
Colorado Department of Education, and the California Dept of Education — which plans to adopt 
assessment measures for its new performance standards in 1997. 
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# Developing a Prototype. During the first year, sample performance tasks and their assessment 
measures will be developed in English/Language Arts, History/Social Science, Mathematics, and 
Science. Through evaluating these first tasks, common features will be identified, barriers and 
successes can be analyzed, and a prototype will start to emerge. Throughout the project, activities 
will be sought that encourage learning by all students, and additional resources identified to ensure 
optional ways to gain competency in information literacy. 

# Evaluating the Prototype. In the second year the prototype(s) will be developed and tested as a 
core of sites will have some experience in using on-line resources and information literacy for 
performance based learning. This prototype will be further tested and revised through Year 3 with a 
small (11) group of additional sites, and will be opened to further testing and refinements during 
Years 4 and 5. 

# Revising the tool(s). From the start a very interactive, user -focused testing/revision process will 
be built into the project. Revision and retesting will be an ongoing, consistent part of each step of 
the process. 

# Collecting Data on Test Validity and Reliability. Experts will conduct evaluations on the project 
and at each cycle evaluation reports will be circulated to professionals and decision makers. At this 
stage, actual achievement data on these assessments must be analyzed and evaluated to determine 
(1) the extent to which students are learning the outcomes and (2) the extent to which the 
curriculum and instruction is effective in facilitating this learning (Freedman, 1993). 

Professional Development . The support of professionals in learning is critical to success. 
What passes for inservice training, and the name evokes memories that make the argument, is 
unlikely to bring about change. Early work on diffusion of innovations demonstrated specific 
strategies are associated with success. These strategies are described in the Rand Studies (Berman 
& McLaughlin, 1975; McLaughlin, 1990) and refined since., identify the important role of the 
adopter in adapting the innovation. These strategies work within the change process as a sequence 
of developmental learning. (Rogers, 1993). Full and collaborative participation is a primary 
determinate of successful dissemination and continuance. 

Development and success of the training will be in the framework of a performance assessment 
matrix for information literacy. The strategy for development will be based on the ACOT (Dwyer, 
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1991) education five step change model. The steps with action added are: 

1. Entry: Identify the volunteer teams, identify innovations to be incorporated into student 
learning; 

2. Adoption: Teaching of the new strategies for telecommunications use, information literacy, 
and performance assessment through professional development; 

3. Adaptation: Applying new knowledge and skills in specific local classroom; 

4. Appropriation: Teachers and students develop their own materials on those principles and 
experiment; 

5. Invention: Balanced use of teaching and project-based learning, integration of alternative 
modes of assessment and resource-based learning. 

Teachers are the key to student attainment of information literacy. Active, student-centered 
learning represents a major shift in instructional strategies, a shift not often addressed in teacher 
preparation; therefore major staff development must be conducted. Teachers and administrators 
must become information literate themselves, comfortable with the variety of resources as well as 
with the process of accessing, evaluating, and using information. Furthermore, the assessments 
developed will need to be integrated into the process with the information literacy approach in 
students' final projects, portfolios, or performance measures. We know this process takes time, 
better measured in years than months, for a teacher to move from technical competence to 
integration of an innovation into classroom practice. (See Honey & Henriquez, 1993.) If credible 
assessment of information literacy skills is important, we must start now to have a significant place 
in the year 2020. 
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